Search AI Products and News

The Sinan OpenCompass team at the Shanghai Artificial Intelligence Laboratory has collaborated with the Modao ModelScope to launch the Compass Multi-Modal Arena, a new section of a large model evaluation platform focusing on multi-modal large models. Users can upload images and input questions to have two anonymous multi-modal large models generate answers, which can then be subjectively evaluated based on the quality of the generated content, allowing users to select the better-performing model. The platform offers an easy-to-use interface and a unique question bank.

Recently, the HELM MMLU released its latest results from Stanford University's large model evaluation leaderboard. Percy Liang, director of the Stanford Center for Research on Foundation Models, stated that Alibaba's Tongyi Qianwen Qwen2-72B model has surpassed Llama3-70B in ranking, becoming the best-performing open-source large model.

["📊 Evaluation System of Large Models: The current evaluation system for large models has issues such as open-source datasets that can be manipulated, fairness problems arising from closed evaluation datasets, and evaluation metrics that are not sufficiently scientific and comprehensive.", "💡 Trend of Large Model Applications: The article mentions that large models have evolved from model-level development to innovation at the application level.", "🔎 Commercialization Issues of Large Models: For large model teams, achieving commercialization is far more important than rankings and parameters." ]

Ant Group, in collaboration with Peking University, has released a benchmark for evaluating large language models in the DevOps field. This benchmark includes a total of 4850 multiple-choice questions across 8 categories such as planning, coding, building, testing, and releasing. The benchmark also provides detailed evaluations for AIOps tasks, showing that the score differences among various models are minimal.

Parameter scale is not the only criterion for assessing large models. Differences in evaluation sets can lead to significant ranking variations. An increase in subjective question proportions can also affect rankings, raising questions about evaluation fairness. Third-party assessment organizations such as OpenCompass and FlagEval are gaining attention. The academic community believes that model robustness, safety, and other dimensions should also be considered. A truly comprehensive and effective evaluation method is still being explored.

SuperCLUE has released the August rankings for Chinese large models, featuring 5 different ranking evaluations that selected 16 general large language models, utilizing 3,337 new test questions. The performance gap between domestic large models on Chinese tasks and GPT-3.5 is narrowing.